{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4da38ad",
   "metadata": {},
   "source": [
    "# 1. Libraries & Sample Data\n",
    "The first step is to load our Python Libraries and download the sample data. The dataset represents Apple stock price (1d bars) for the year 2010"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa05430",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Python Libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "# for dataframe display\n",
    "pd.set_option(\"display.max_rows\", None)\n",
    "def display_df(df):\n",
    "    # Puts the scrollbar next to the DataFrame\n",
    "    display(HTML(\"<div style='height: 200px; overflow: auto; width: fit-content'>\" + df.to_html() + \"</div>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa831031",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download Sample Data\n",
    "data = pd.read_csv('AAPL_2009-2010_6m_all_features_1d.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd9233ca",
   "metadata": {},
   "source": [
    "# 5. State Space Representation\n",
    "Now we have a set of data with OHLC data plus some techinchal indicators. Using this data, construct the state space matrix, whese features of the state space are Close Price, 5-day Moving Average, 20-day Moving Average, Bollinger Bands (upper and lower), and 20-day Historical Volatility of Close Price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "350ef137",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct the State Space Matrix\n",
    "dataset = data.reset_index()[['Date', 'Close', 'MA5', 'MA20', 'BB_upper', 'BB_lower', 'Vol20']]\n",
    "display_df(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e573f39",
   "metadata": {},
   "source": [
    "# 6. Z-Score Normalization\n",
    "Now that we have cleaned our data, and created our features of interest, we must normalize our data. For this example, we use the sklearn StandardScaler, which centers the data and normalizes to unit variance (i.e. performs z-score normalization for us). Do this in a simple, non-rolling fashion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aab1517",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display & Plot Un-normalized Dataset\n",
    "display_df(dataset)\n",
    "dataset['Close'].plot()\n",
    "dataset['BB_upper'].plot()\n",
    "dataset['BB_lower'].plot(rot=45)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "588b508b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize Dataset with StandardScaler\n",
    "normlist = []\n",
    "static_normed_dataset = pd.DataFrame(index=dataset.index)\n",
    "for col in dataset.columns:\n",
    "    if col == 'Date':\n",
    "        static_normed_dataset[col] = dataset[col]\n",
    "        continue\n",
    "    normalizer = StandardScaler()\n",
    "    column_data = pd.DataFrame(dataset[col])\n",
    "    normalizer.fit(column_data)\n",
    "    static_normed_dataset[col] = normalizer.transform(column_data).flatten()\n",
    "    normlist.append(normalizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad5c80b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, MA20, BB Upper, BB Lower\n",
    "static_normed_dataset['Close'].plot()\n",
    "static_normed_dataset['MA20'].plot()\n",
    "static_normed_dataset['BB_upper'].plot()\n",
    "static_normed_dataset['BB_lower'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88d8d6d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, MA20, MA5\n",
    "static_normed_dataset['Close'].plot()\n",
    "static_normed_dataset['MA20'].plot()\n",
    "static_normed_dataset['MA5'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10e03310",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, Volatility\n",
    "static_normed_dataset['Close'].plot()\n",
    "static_normed_dataset['Vol20'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2f718a0",
   "metadata": {},
   "source": [
    "# 7. Rolling Z-Score Normalization\n",
    "Now that we have cleaned our data, and created our features of interest, we must normalize our data. For this example, we use the sklearn StandardScaler, which centers the data and normalizes to unit variance. Due to the fact that our data is time-series data, it is best practice to do this in a rolling fashion. We choose 20 days as our window for normalization, and run the StandardScaler in a rolling (non-overlapping) fashion. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b491d9fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display raw dataset (unnormalized) \n",
    "display_df(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f2cbb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize the chosen price data & features\n",
    "normed_dataset = pd.DataFrame(index=dataset.index, columns=dataset.columns)\n",
    "step = 20\n",
    "for col in dataset.columns:\n",
    "    n = 0\n",
    "    if col == 'Date':\n",
    "        normed_dataset[col] = dataset[col]\n",
    "        continue\n",
    "    while n<=len(data.index):\n",
    "        normalizer = StandardScaler()\n",
    "        if n == 0:\n",
    "            column_data = dataset.loc[:n+step, col]\n",
    "            normalizer.fit(column_data.values.reshape(-1, 1))\n",
    "            normed_dataset.loc[:n+step, col] = normalizer.transform(column_data.values.reshape(-1, 1)).flatten()\n",
    "        elif n+step >= len(data.index):\n",
    "            column_data = dataset.loc[n:, col]\n",
    "            normalizer.fit(column_data.values.reshape(-1, 1))\n",
    "            normed_dataset.loc[n:,col] = normalizer.transform(column_data.values.reshape(-1, 1)).flatten()\n",
    "        else:\n",
    "            column_data = dataset.loc[n:n+step, col]\n",
    "            normalizer.fit(column_data.values.reshape(-1, 1))\n",
    "            normed_dataset.loc[n:n+step, col] = normalizer.transform(column_data.values.reshape(-1, 1)).flatten()\n",
    "        n += step\n",
    "        # normlist.append(normalizer)\n",
    "display_df(normed_dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dcab016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, MA20, BB Upper, BB Lower\n",
    "normed_dataset['Close'].plot()\n",
    "normed_dataset['MA20'].plot()\n",
    "normed_dataset['BB_upper'].plot()\n",
    "normed_dataset['BB_lower'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd7c59e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, MA20, MA5\n",
    "normed_dataset['Close'].plot()\n",
    "normed_dataset['MA20'].plot()\n",
    "normed_dataset['MA5'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "678b843f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Normalized Features: Close, Volatility\n",
    "normed_dataset['Close'].plot()\n",
    "normed_dataset['Vol20'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58c53e80",
   "metadata": {},
   "outputs": [],
   "source": [
    "normed_dataset.to_csv('AAPL_2009-2010_6m_normed_1d.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rl-project-udacity",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
